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(54) Detecting endoscope insertion direction 

(57) The direction for further insertion of an endoscope on the basis of the form of a fold existing on the inside wall of part of 
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form a further image, the further image b divided Into a plurality of images, contour segments based on the discontinuous 
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or colour of the image varies. Extracting the contour segments may comprise of extracting the optimum contour segment 
from among a plurality of extracted candidates. A fibre optic endoscope with a curvabte end section may be used. 
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METHQPS 9P PBTEPHlfS BttPQSqpPE IffgffiTIQH prRgpnoq; 

This invention relates to methods of detecting the 
requisite direction of insertion of an endoscope, and more 
particularly to an endoscope insertion direction detecting method 
adapted to automatically aid the insertion of an endoscope in the 
large Intestine, to facilitate medical inspection. 

Recently, there has been extensive use of an endoscope 
whereby organs within a body cavity can be observed by inserting an 
elongate Insertion sheath into the body cavity and various curing 
treatments can be made by using treatment tools, inserted through 
the sheath in a treating tool channel, as required. 

In a conventional endoscope inspection, the doctor judges 
the required direction for advance of the endoscope insertion 
sheath by observing the endoscope image, while inserting the 
endoscope. 

However, high levels of technique and skill are required 
to insert an endoscope in inspecting the large intestine. 

One object of the present invention is to provide an 
endoscope insertion direction detecting method whereby the 
endoscope insertion direction can be simply and quickly detected, 
whilst the adverse influence of noise can be reduced, enabling the 
appropriate endoscope insertion direction to be detected more 
accurately. 



In accordance with the present invention a method of 
detecting a required endoscope insertion direction includes 
extracting the form of a fold existing on the Inside wall of an 
observed part from an endoscope picture image and to judge the 
endoscope insertion direction on the basis of the form of the fold 
extracted by this step. It comprises / for example, a step of 
extracting discontinuous points at which the brightness or colour 
in an endoscope picture image varies. 

Advantageously, after extracting said discontinuous 
points, the invention comprises the further steps of -extracting 
contour segments based on said discontinuous points from respective 
divided picture images obtained by dividing the picture image 
obtained by the above-mentioned step of extracting the 
discontinuous points into a plurality of picture images, a step of 
connecting the contour segments obtained by the above-mentioned 
step of "extracting the contour segments, and a step of considering 
the line connected by the above-mentioned connecting step to be the 
form of a fold existing on the inside wall of the observed part and 
judging the endoscope insertion direction on the basis of this form 
of the fold. The above-mentioned step of extracting the 
discontinuous points determines the degree to which the brightness 
of the picture Image varies. The above-mentioned step of 
extracting the contour segments has, for example, a step of 
extracting contour segment candidates and a further step of 
extracting the optimum contour segment from among those 
candidates. For example, the above-mentioned step of extracting 
the candidates includes using a modified Hough transformation in 
extracting the contour segments. 



i 



3 

The above-mentioned step of extracting the optimum contour segment 
includes making a perceptual grouping to extract the optimum 
contour segment. The above-mentioned connecting step includes, for 
example, extracting a search starting contour segment by utilising 
a pyramid quad tree structure. 

The invention will now be described with reference to the 
drawings, in whlch:- 

Pigure 1 is a flow chart showing the steps of one 
exemplary method in accordance with the .present invention; 

Figure 2 Is an explanatory view representing the 
Insertion sheath of an endoscope entering into the large intestine; 

Figure 3 is a perspective view showing the tip of the 
endoscope Insertion sheath; 

Figure 4 is an explanatory view showing the Insertion 
sheath of an endoscope advancing into a bent part of the large 
intestine; 

Figure 5 is an explanatory view showing the endoscope 
image obtained when in the state shown in Figure 4; 

Figure 6 is an explanatory view showing the Insertion 
sheath of an endoscope entering into a straight part of the large 
intestine; 

Figure 7 is an explanatory view showing the endoscope 
image obtained when in the state shown in Figure S; 

Figure 8 is an explanatory view showing an example of a 
suitable endoscope apparatus, using a fibre-scope and externally 
fitted television camera; 



Figure 9 is an explanatory view showing an example of a 
suitable endoscope apparatus using a video-scope; 

Figure 10 is a view for explaining the use of a spatial 
filtering in the first step; 

Figure 11(a) is an explanatory view shoving an overlapped 
matrix determining a gradient in an x-dlrectlon, and 

Figure 11(b) is an explanatory view showing an overlapped 
matrix determining a gradient in a y-direction; 

Figure 12 is an explanatory view showing an exemplary 
picture image, as obtained in the first step, when divided into 
small squares of 8 x 8 pixels; 

Figure 13 is an explanatory view shoving a small square 
of 8 x 8 pixels; 

Figure 14 is an explanatory view showing a contour 

segment 

of a fold located in a small region of 8 x 8 pixels; 

Figure 15 is an explanatory view showing a small square 
of 8 x 8 pixels with addresses applied around the outer periphery 
to make a modified Hough transformation; 

Figure 16 is an explanatory view shoving arranged 
elements obtained by a modified Hough conversion of straight lines; 

Figure 17 is an explanatory view showing arranged 
elements obtained by modified Hough conversion of straight lines 
and an example of a number of edge points located on straight lines 
corresponding to the respective arranged elements; 

Figure 18 is a table shoving straight lines of a large 
number of edge points; 

Figure 19 is an explanatory viev shoving contour segments 
vithln a small square; 



Figure 20 is a table shoving grouping by an edge 
orientation; 

Figure 21 is a table shoving grouping by a continuity; 

Figure 22 is an explanatory viev showing edge points on a 
small square for explaining grouping by a continuity; 

Figure 23 is a table shoving an example of a result of a 
perceptual grouping; 

Figure 24 is an explanatory viev shoving a pyramid quad 
tree structure; 

Figure 25 is a partial flow chart shoving details of the 
fourth step; 

Figures 26(a) to (d) are explanatory views shoving small 
squares to be searched next in the case of connecting contour 
segments; 

Figures 27(a) to (d) are explanatory views showing a 
method of determining an insertion direction from the form of a 
fold; 

Figure 28 is a partial flow chart shoving a step of 
determining the centre of gravity of a ring; 

Figure 29 Is an explanatory viev shoving an ellipse of a 

fold; 

Figure 30 is an explanatory view shoving a step of 
determining an x-coordinate of the centre of gravity of a ring; and 

Figure 31 is an explanatory viev shoving a step of 
determining a y-coordinate of the centre of gravity of a ring. 

Figures 32 to 34 relate to the second embodiment of the 
present invention. 
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Figure 32 is an explanatory view shoving a part of an 
overlapped pyramid structure. 

Figure 33 is an explanatory view showing a connection of 
contour segments in an overlapped pyramid structure* 

Figure 34 is a flow chart showing the fourth step of the 
embodiment* 

First of all, a summary of the present invention will be 
given by way of explanation with reference to Figures 2 to 7. 

As shown in Figure 2, an endoscope 1 of the fibre-scope 
type is provided with an elongated flexible insertion sheath 2 
connected to extend from the rear of a thick operating part 3. A 
flexible optical cable 4 extends from a side-port of the operating 
part 3 and is provided at its end with a connector 5, which can be 
connected to a light output port of an illuminating source 6 
containing a lamp 6a. The operating part 3 is provided at its rear 
end with an eye-piece 8. 

As shown in Figure 3, a rigid tip 11 is supported on a 
curvable section 12 provided at the remote end of the insertion 
sheath 2. The operating part 3 is provided with a curvature 
control operating knob (not illustrated) so that the curvable 
section 12 may be curved vertically and/or horizontally, as 
required, by operation of this curvature control operating knob. 

An illuminating lens 15 of an illuminating optical system 
and an objective lens 16 of an observation optical system are 
arranged In the sheath end-face directed substantially in the same 
direction as the axis of the rigid tip 11. A light guide (not 
illustrated), for example an optical fibre bundle, is provided In 



the sheath 2 at the rear of the illuminating lens 15. This light 
guide is inserted through the insertion sheath 2, operating part 3 
and optical cable 4, and is connected to the connector 5 so that, 
when this connector 5 is coupled to the light source 6, the 
illuminating light emitted by the lamp 6a enters the light guide, 
to be led to the tip 11 and emitted from the tip end-face to be 
radiated out to an object through the illuminating lens 15, as 
indicated in Figure 3 by solid lines 17 representing the 
illuminated region. 

The end -face of an image-relaying fibre bundle (not 
illustrated) is arranged in the image-forming plane of the 
objective lens 16. This image guide is inserted through the 
insertion sheath 2 and extends to the eye-piece 8. The object 
image formed by the objective lens 16 will be led to the 
eye-piece 8 so that it can be observed through the eye-piece lens 
{not shown), which is mounted in the eye-piece 8. In Figure 3, a 
pair of broken lines 18 indicate the visual field of the observing 
optical system. 

Now, as shown in Figure 3, the illuminating optical 
system and observing optical system of the endoscope 1 are adjacent 
to each other and axe directed substantially in the same direction. 

On the other hand, annular folds, also called haustras, 
exist on the inside wall of the large intestine. Most endoscope 
doctors judge the endoscope insertion direction by the manner in 
which such an annular fold is seen. That is to say, the centre of 
the ring of such fold is an excellent criterion in Judging the 
endoscope insertion direction. This shall be explained with 
reference to Figures 4 to 7. 
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In Figures 5 and 7, folds 21 and 22 are represented on 
the inside wall of the large intestine. 

Figure 4 shows the case when inserting the insertion 
sheath 2 of the endoscope 1 into an upwardly bent part of the large 
intestine. In such a case, as shown in Figure 5, the folds are 
seen as being upwardly deviated. Therefore, in this case, the 
tip 11 of the endoscope 1 should be curved upward and the insertion 
sheath 2 will then be inserted to move upwardly. 

Figure 6 show the case of inserting the insertion 
sheath 2 of the endoscope 1 into a straight part of the large 
intestine 20. In this case, as shown in Figure 7, the folds are 
seen with no deviation, vertically or horizontally. Therefore, in 
this case, the insertion sheath 2 of the endoscope 1 may be 
inserted in its straight condition. 

Thus, in the proposed methods of endoscope insertion 
direction detection, the form of the folds existing on the inside 
wall, as seen in the endoscope image, is extracted and the 
endoscope insertion direction is deduced on the basis of the form 
of these folds. 

This embodiment of an endoscope insertion direction 
detecting method can be employed with endoscopes such as are shown, 
for example, in Figure 8 or Figure 9. 

The endoscope apparatus shown in Figure 8 is provided 
with a fibre-scope 1 fed with illuminating light by a light 
source 6, and an externally fitted television camera 30 is fitted 
to the eye-piece 8 of this fibre-scope 1« The construction of the 
fibre-scope 1 is the same as that shown in Figure 2 and need not be 
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farther explained here. The externally fitted television camera 30 
is provided, for example, with an image lens (not illustrated) that 
forms an image of light from the eye-piece 8 and utilises a solid 
state imaging device (not illustrated), arranged in the image plane 
of this image-forming lens. This externally fitted television 
camera 30 drives the solid state Imaging device and is connected to 
a signal processing apparatus 31 to convert the output signal of 
this solid state imaging device to form a video signal. The video 
signal output of the signal processing apparatus 31 is fed into a 
monitor 32, and also converted to digital form by an 
A/D-converter 33, which then feeds into an electronic computer 35, 
and can be taken into a memory (not illustrated) within this 
electronic computer 35. The endoscope image will be displayed on 
the monitor 32, and the endoscope insertion direction detecting 
method in this embodiment can be carried out by the electronic 
computer 35. 

The endoscope apparatus shown in Figure 3 is provided 
with a video-scope 41 instead of a fibre-scope 1 and externally 
fitted television camera 30; As with the fibre-scope 1, this 
video-scope 41 is provided with an elongate flexible insertion 
sheath 2 and an operating part 3 connected to the rear of this 
insertion sheath 2. A flexible optical cable 42 extends sidewise 
from the operating part 3 and is provided at its end with a 
connector 43 which is connected to a control apparatus 45 
containing a light source 6 and a video signal processing 
circuit 46. A solid state imaging device (not illustrated) is 
arranged in the image forming position of the objective lens in the 
tip of the insertion sheath 2 of the video-scope 41 and Is 



10 

connected to a video signal processing circuit 46 within the 
control apparatus 45 through the signal lines inserted through the 
insertion sheath 2, operating part 3, cable 42 and connector 43. 
The illuminating optical system of the video-scope 41 is the same 
as that of the fibre-scope 1, in that the illuminating light 
emitted from the lamp 6a of the light source 6 within the control 
apparatus 45 may enter the light guide at the entrance end. The 
solid state imaging device will be driven by the video signal 
processing circuit 46, and the output signal of this solid state 
imaging device will be processed to form a video signal by the 
video signal processing circuit. As in the endoscope apparatus 
using the fibre-scope 1, the video signal output from this signal 
processing circuit 46 can be fed into the monitor 32, and can be 
converted to digital form by the A/D-converter 33, to be input into 
the electronic computer 35 and taken into the memory (not 
illustrated) within this electronic computer 35. The endoscope 
image will be displayed on the monitor 32 and the endoscope 
Insertion direction detecting method in this embodiment will be 
carried out by the electronic computer 35. 

The endoscope insertion direction detecting method 
utilised in this embodiment will be further explained in the 
following, with reference to Figure 1. 

As indicated in the flow chart shown in Figure 1, this 
direction detecting method comprises a first step SI, extracting 
discontinuous points in an original image fed into the electronic 
computer 35, a second step 32 dividing the picture image obtained 
in the first step SI into a plurality of picture images and 
extracting contour segment candidates by using a modified Hough 
conversion from each of the divided picture Images, a third step S3 
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making perceptual grouping of the contour segment candidates 
obtained in the second step and extracting the optimum contour 
segment from each of the divided picture images, a fourth step S4 
connecting the contour segments obtained in the third step S3, and 
a fifth step 35 for determining the Insertion direction from the 
result of the 
fourth step S4. 

The first step will be explained with reference to 
Figures 10, 11a and lib. 

In extracting discontinuous points, a colour defined by 
red, green and blue values may be noted, or a grey level (thickness 
or brightness}. In this embodiment, the case of noting a grey 
level will be explained. The number of pixels of the original 
picture shall be 512 x 512 and the grey level shall be of 256 
gradations. 

Extracting discontinuous points by noting a grey level is 
achieved by inspecting the variation rate (gradient) of the grey 
level on a spatial coordinate, and extracting the grey level in the 
variation. This is an edge detection by noting the grey level. 

For the edge detecting, for example, spatial filtering 
using a overlapped matrix is used in this embodiment. 

Spatial filtering for the case using an overlapped 
matrix consisting of 3 x 3 pixels will be explained with 
reference to Figure 10. In Figure 10, Pi (xi,yi) 
represents the grey level of the pixel of an Input picture 
Image Pi having a coordinate (x|,yi). In the same 
manner, P2 (xi,yi) represents the gradient of the picture 
image of coordinate (xi,yi) in an output picture image P2. 
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First of all, the locality of 3 x 3 in the input picture 
image Pi (xi,yi) is taken out, a product of the value of each 
pixel in this local zone of 3 x 3 calculated, and the value of each 
element corresponding to the overlapped matrix consisting of 
separately prepared 3x3 elements is calculated, to determine a 
sum of 9 products for the zone ?2 (x£,yi) in the output 
picture image. 

This operation is repeated in turn for the respective 
pixels of the input picture image to obtain an output picture 
image P2 to which spatial filtering has been applied. 

How, a gradient (grey level variation rate) g x in 
the x-direction is obtained by using an overlapped matrix as shown 
in Figure 11(a). In the same manner, a gradient (grey level 
variation rate] g y In the y-direction is obtained by using an 
overlapped matrix as shown in Figure 1Kb). An absolute value of 
the gradient in a pixel is given by the undermentioned 
formula (1-1) but, in this embodiment, the absolute value need not 
be very accurate and therefore, for brevity in the operation, it 
may be approximated by a formula (1-2) as follows:- 

g = gx2 + gy2 (1-1). 

g * Igx! + Igyi u-2). 

where g represents the magnitude of the discontinuity. 
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For the pixel of the coordinate (xi,yi) in the case 
of the overlapped matrices shown in Figures 11(a) and 1Kb), g x 
and gy are used, and are given in concrete terms by the following 
formulae, (1-3) and (1-4):- 

gx = £>2 Ui/Yi) 

= -Pi (xi - 1/ yi + i) 
+P1 (xi ♦ i, yi + i) 
--[T • pi (xi - i, yi) 
♦V^" ■ pi (xi + i + yi) 
-Pl (xi - i, yi - i) 

+Pi (xi + i + yi - i) (1-3). 

gy * P2 (xi,yi) 

a pi (xi - i, yi + l) 

+t^T" • p i {x i + yi + i) 
+pi (xi + i, yi + D 
-Pi (xi - i + yi - i) 
-VT • pi (xi, yi - i) , 

-Pl (xi + i t yi - i) (1-4). 
The direction of the edge is given by the following 
formula (1-5):- 

O = arctan (gy/gx) (1-5). 
Here, g is determined by the formula (1-1) or (1-2), and 
is compared with a predetermined reference value g r , and edge 
points above the reference value g £ are not kept. 
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To reduce the number of discarded edge points one conld 
set the reference value g r to be a large value, but If the 
reference value g r is set to be too large a value, then 
inherently necessary edge points are likely to be discarded, and 
therefore it is important to set the value g c to be rather low. 
It Is preferable to set the reference value g r so as to leave 
about 80% of all the edge points. It may be set so that about 60% 
of the edge points remain after the dispersion of g of all the edge 
points is determined. 

Thus, when the somewhat small value of g r is used and 
the perceptual grouping Is then made, a weak but significant edge 
point can be extracted without being Influenced by noise or the 
like. This is one of the advantageous features of this embodiment. 

The second step will now be explained with reference to 
Figures 12 to 19. 

First of all, as shown in Figure 12, the output picture 
image 92 l*i,Yi) obtained in the first step is divided 
into small squares of about 8x8 pixels and is converted in a 
modified Hough manner. Where an original picture image consists 
of 512 x 512 pixels. It will be divided Into 64 x 64 picture 
images. Thus, in this embodiment, it is divided into small squares 
of 8 x 8 pixels by way of example, but, depending on the required 
precision, It may be divided into 4x4 pixels, or other values as 
desired. 
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An output picture Image P2 (xi,yi) and a selected 
small square of 8 x 8 pixels are shown respectively In Figures 12 
and 13. As shown in Figure 14, If a small square of from 4x4 
to 16 x 16 pixels is selected/ through which the fold passes, the 
short length of fold within the snail square can be considered as a 
substantially straight line. 

First of all, we should briefly explain the modified 
Hough conversion* 

Figure 13 is a simplified representation of a common, 
well used x,y coordinate system. In order to make a modified Hough 
conversion, then as shown in Figure 15, addresses are attached all 
round the outer periphery of the small square of 8 x 8 pixels. 

Then, the straight line on the small square of 8 x 8 
pixels can be defined by designating a start address (S) and end 
address (E). For example, the straight line represented by a 
broken line in Figure 15 can be defined as a straight line having a 
start address 3 and an end address 20. 

On the other hand, the kinds of straight lines which 
can be described on the small square of 8 x 8 pixels 
are C 32 x 32)/2 = 512. The reason for multiplying (32 x 32) 
by 1/2 is that the straight line on the small square is not a 
vector. That is to say, the straight line between the start 
address 3 and end address 20 can be considered to be identical to 
the straight line between the start address 20 and end address 3. 

Therefore, all the straight lines correspond to one of 
the elements as arranged in Figure 16. 
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When any edge points on the straight line corresponding 
to one arrangement on the arranged elements are present within any 
one of the arranged elements, for example, considering the straight 
line shown as a broken line in Figure 15, a straight line having a 
start address 3 and an end address 20, it passes through four small 
squares having four edge points, and the line will therefore be 
expressed as in Figure 16. 

If a modified Hough transformation in accordance with 
such an idea, in fact, a result such as is shown in Figure 17 will 
be 

obtained* As already described, one of the arranged elements 
Cone of the squares In Figure 17} corresponds to one straight line 
and the numeral (vote) within the arranged element represents the 
number of edge points existing on that straight line. 

The larger the number of edge points existing on any 
straight line, the higher is the possibility that this is in fact 
the required straight line. 

Therefore, about 5 votes are extracted as contour segment 
candidates from the larger number of votes. Such extracted contour 
candidates bears a mark O *n Figure 17 and are shown as a table in 
Figure Id. 

Here lies an idea of extracting the largest number of 
votes and making it a required contour segment. However, if it is 
made so, a wrong contour segment is likely to be extracted, for 
example, as In the case shown in Figure 19 in which (a) is a 
contour segment to be extracted, whereas, if judged only by the 
number of votes, the contour segment (b) will be erroneously 
extracted because of noise content. 
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In order to avoid such danger, the number of votes is 
reduced to about 5 arranged elements, any constant grouping is then 
identified, and the optimum contour segment can then be extracted * 

The perceptual grouping of the third step will now be 
explained with reference to Figures 20 to 23. 

In this embodiment, the perceptual grouping is made by 
noting the three items defined below. However, in addition to the 
under -mentioned three items, the size of the gradient (edge 
magnitude) may be noted, or the colour may be noted. The items 
are:- 

1- Edge orientation 
(on edge points} : 
Reference: ± 22. 5° 

2. Grey level 
(on pixels) : 
Reference: d= 4 level 

3. Continuity 

(on edge points}: 
Reference: ± 1 pixel distance 
Bach of the above-mentioned three items will now be 

explained. 

For example, in the case of Figure 17, the following 
processes are made on each of the five lines in Figure 17:- 
1. Sdoe orientation:- 

For example, the line 4, the arranged element (16,4) in 
Figure 18 has 9 edge points arranged in the order of the smaller 
edge direction 0 determined in the first step, as shown in 
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Figure 20. A new grouping is made where there is a gap larger 
than 450. In the examples shown in Figure 20, the difference 
of 0 between tbe 8th edge point and 9th edge point is: 

620 - 350 - 270 (> 22. 50 J.- 
Here, a regrouping is made. 
2L* Grey 

As in the case of the above-mentioned edge orientation, 
the grey level of the parts corresponding to the edge points are 
arranged in the order of the smaller one and a grouping is made 
where the gap of the grey levels is larger than 4. 
3- Continuity:- 

The x-coordinates and y-coordinatesof the edge points are 
noted and are arranged in the order of the smaller x-coordinate as 
shown in Figure 21. The same x-coordinates are arranged in the 
order of the larger y-coordinate. Where the differencedx of the 
x-coordinate is Ax > 1 or the difference Ay of the y-coordinate 
is Ay > 1, a grouping is made. In the example shown In Figure 21, 
the' difference of the y-coordinate between the 4th edge point 
and 5th edge point is 2 and here a grouping Is made. Thus, by 
noting both of the x-coordinate and y-coordinate, even if the 
straight line rises extremely or lies level with respect to the 
x-axis or y-axis, the discontinuous point can be positively 
extracted and a grouping can be made. For example, in Figure 22, 
the respective edge points in the example in Figure 21 are plotted 
on the x and y-coordinates. In such a case, if only the 
x-coordinate is noted, it is not possible to detect that the 
straight line is discontinuous. 
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By making the above three operations, generally such 
results, as are shown for example In Figure 23, are obtained. 

Here, for example, if a logical formula of 

Continuity O (Grey LevelJH (Edge Orientation) (1) 
is applied as a condition of extracting the optimum contour 
segment, as shown in Figure 23, the edge points can be grouped in 
six groups of A to F. In the example in Figure 23, the group A has 
the most edge points and the number of the edge points is 6. 

The condition of extracting the optimum contour segment 

is 

not limited to the logical formula (1) but the following logical 
formula (2) may be used, for example:- 

Continuity O (Grey Level U Edge Orientation) (2). 

The same is carried out also on the other arranged 
elements and the group having the most edge points among them is 
extracted and forms a contour segment to be extracted in the small 
square of 8 x 8 pixels ♦ Thus, in the third step, a contour segment 
of 64 x 64 small regions consisting of 8 x 8 pixels could be 
extracted. (Heedless to s^y, there are many small regions In which 
no contour segment exists). The division into small regions gives 
advantages, in that parallel processes by a plurality of computers 
are possible and the operating time can be reduced. Parallel 
processes may be made by using a plurality of exclusive ICs. 

Furthermore, also in the case of noting the size of the 
gradient (edge magnitude) or the colour, in the same manner, the 
edge points may be arranged in the order of the size of the 
gradient or the colour and may be grouped where there is a gap 
larger than that predetermined. 
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The fourth step will now be explained vith reference to 
Figures 24 to 26. 

In this fourth step, the contour segments obtained In the 
third step are connected. This is called a tracing or connecting 
of edges. 

In tracing the edges, that segment from which the tracing 
is bo be started is important. In this embodiment, a pyramid quad 
tree structure is utilised to trace the edges, k step of obtaining 
a curve of a fold by utilising this pyramid quad tree structure 
will now be explained with reference to Figures 24 and 25. 

First of all, as shown in Figure 24, a pyramid quad tree 
structure is made by making a small region, consisting of 8 x 8 
pixels, a leaf node (or leaf). That is to say, in the step S41 
shown in Figure 25, a small region of 8 x 8 pixels is made a leaf 
node and, in the step S42, the sum of the contour segment numbers 
of four children nodes is made a contour segment n amber of a parent 
node to obtain a picture image of n/2 x n/2 pixels. Then, through 
the step S43 of Judging whether the number of pixels is 1 or not, 
the step S42 is repeated until a root (or root node) is reached. 
In the thus made pyramid quad tree structure, the parent node 
retains the number of the contour segments of the children nodes. 
In Figure 24 the figures near the respective nodes represent the 
numbers of contour segments. 

Then, in order to discover a starting contour segment, 
the tree is searched downward from the root, that is to say, in the 
step 544, the operation of selecting that child node having the 



most contour segments among the four son modes is repeated. As 
shown in Figure 24, in the stage of the leaf, in a case where a 
plurality of leaves having contour segments exist , any contour 
segment may be made a starting contour segment. 

Then, in the step 545, the contour segment determined in 
the step S44 is made a starting contour segment and the contour 
segments are connected. 

The operation of connecting these contour segments will 
now be explained with reference to Figures 26(a) to (d). 

As shown in Figure 26(a), in case the contour segments 
are connected in the order of the small regions 1 and 2, the next 
searching direction will he the direction indicated by the arrow in 
the drawing. Therefore, in this case, the small regions a, b and c 
are checked. 

In the same manner, as shown in Figure 26(b), in case the 
contour segments are connected in the order of the small regions 1 
and 2, the next searching direction will be the direction indicated 
by the arrow in the drawing. In this case, the small regions a, b 
and c are checked. 

As shown In Figure 26(c), in case the contour segments 
are connected in the order of the small regions 1 and 2, the small 
regions a, b and c are checked but, in case the contour segments 
exist in both of the small regions a and c, the directions of the 
contour segments a and c are checked and the direction in which a 
smooth connection is made is selected. In the case of Figure 
26(c), the contour segment of the small region c is selected. 
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As shown In Figure 25(d), In case the contour segments 
are connected in the order of the small regions 1 and 2 and no 
contour segment exists In the small regions a, b and c, the small 
regions d, e, f, g, b, 1 and j are inspected, because, in case the 
contour segment a is considered to have lapsed for any reason, the 
next search will be of d, e and f , In the same manner. In case the 
contour segment b is considered to have lapsed, the next search 
will be of f, g and h and, in case the contour segment c is 
considered to have lapsed, the next search . will be of h, i and j. 

In case the contour segments are to be connected, the 
angle formed by two contour segments to be connected may be limited 
to be within±450. 

Whenever the contour segments are connected, the 
numerical value on the quad tree will be corrected * That is to 
say, the 

value of the leaf corresponding to the connected contour segment 
will be changed from 1 to 0. The value of the parent node above it 
will also be corrected. , 

The contour segments are connected in both directions* 

When a curve is thus obtained, in the step S46, whether 
the fourth step is to be concluded or not is judged and, in case It 
is not be be concluded, when the steps in and after the step S41 
are again repeated, another curve can be obtained. 

The fifth step shall be explained In the following with 
reference to Figures 27 to 31. 

In the fifth step, the insertion direction is judged and 
determined by the form of the fold determined in the fourth step. 
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The form of the fold to be determined is of such pattern 
as is shown, for example, in Figures 27(a) to (d). The insertion 
direction is determined respectively by the methods explained in 
the following. 

Figure 27(a) shows the case that two carves (folds) C, 
Ci and C2, having an intersection (or branching point) 
indicated by the mark * in the drawing are obtained. In this case, 
by judging the direction of the contour segment at the point 
indicated by the mark * in the drawing, the inner fold is judged. 
In the 

case of Figure 27(a), the curve C2 is inner and the centre 
(point a) of the inner curve is made an insertion direction. In a 
case where such a curve is obtained, it is considered to be a case 
where such a tube cavity of the large intestine in which an 
endoscope is inserted is curved, and the inner fold is partly 
hidden. 

In the method shown in Figure 27(b), radial lines are 
constructed respectively at about five points on the obtained curve 
and the part in which the intersections of the radial lines 
concentrate is made the insertion direction. 

In the method shown in Figure 27(d), in any case where 
the obtained curve is broken anywhere, the nearest curves are 
connected and the centre of gravity of the obtained ring is made an 
Insertion direction. 

In the methods shown in Figures 27(b) to (d), any curves 
or rings may be used, the largest curve or ring may be used or the 
ring n-th from a large ring may be predetermined to be used. 
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The centre of gravity can be determined by such a process 
as is shown, for example, in Figure 28. 

First of all, in the step S51, the number of pixels 
contained in the circle or ellipse of a fold is determined and is 
made N as shown in Figure 29. The number of pixels may be replaced 
with the number of small squares used in the step. 

Then, in the step S52, as indicated by the arrows in 
Figure 30, the number of pixels contained in the circle or ellipse 
of the fold is counted in the direction of the x-axis from above 
until the number becomes N/2. The value of the y-coordinate when 
the number becomes N/2 is represented by y 0 which is a 
y-coordinate of the determined centre of gravity. 

In the same manner, in the step S53, as indicated by the 
arrows in Figure 31, the number of pixels contained in the circle 
or ellipse of a fold is counted in the direction of the y-axls from 
the left side until the number N/2. The value of the x-coordinate 
when the number becomes N/2 is represented by *q which is an 
x-coordinate of the determined centre of gravity. 

Then, in the step S54, the centre of gravity of the fold 
is determined as (x 0 ,y 0 ). The form of the fold has been 
described as circular or elliptical, but is not limited to such 
shapes . 

As the endoscope is inserted, the form of the fold trill 
vary. Therefore, the centre of gravity of the fold n-th as counted 
from a large one may always be determined and may be judged to be 
an insertion direction or an average value of the centre of gravity 
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of the n-th fold as counted from a large one and the centre of 
gravity of the n + m-th fold may be determined and Judged to be an 
insertion direction. Also, the centre of gravity of the fold on 
the innermost side may be made an insertion direction. The 
direction in which the most centres of gravity among the centres of 
gravity of a plurality of folds are located may be made an 
insertion direction, 

As shown in Figure 4, in a case where the large 
intestine 20 is curved, then as shown in Figure 5, the centre of 
gravity of the fold is different depending on the fold. In the 
case of Figure 5, the more the centre of gravity is on the Inside 
of the fold, the more the insertion direction side is on the upper 
side. The larger the curvature, the larger the displacement of 
this centre of gravity. Therefore, the curvature of the large 
intestine 20 can be detected from the displacements of the centre 
of gravity of a plurality of folds. 

Thus, according to this embodiment, the form of a fold is 
determined by the first to fourth steps, the endoscope insertion 
direction is judged in the fifth step on the basis of the form of 
the fold and thereby the endoscope insertion direction can be 
simply detected* 

In the case of extracting a discontinuous point 
(edge point) in the endoscope picture by the first step, when the 
reference value Is set to be rather low and the value of the 
gradient g is left to be small to some extent, a necessary 
discontinuous point (edge point) will be able to be extracted 
without being Influenced by noise or the like. 
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la cases where contour segment candidates are extracted 
by utilising a modified Hough conversion in the second step or in 
cases where the optimum contour segment is extracted from the 
respective divided picture Images by a perceptual grouping in the 
second step, then by dividing the endoscope picture image into 
small regions, parallel processing by a plurality of computers is 
made possible and the operating time can be reduced* 

As a pyramid quad tree structure is utilised in 
extracting a starting contour segment in the fourth step, the 
processing time can be remarkably reduced. 

Incidentally, in this embodiment/ a grey level is noted 
In extracting discontinuous points in the first step, but a colour 
may equally be noted, as described above. 

In case a colour is noted, for example, the variation 
rates of- the hue and saturation may be Inspected and the hoe and 
saturation as varying may be extracted. 

For example, in case three primary colour components 
(three exciting values) R, G and fi of a CIE-RGfi colouring system 
are obtained from an original picture, a hue 0 can be represented 
by a formula (2-2), by using the following formula (2-1):- 
_ 2r - g - b 

©1 = cos-1 (2-1) 

{6 {(r-l/3)2 + (g-l/3)2 + (b-l/3)2) 

0° e i(g£b), 6= 2v-6i (g < b) (2-2) 
where: r = R/{R + G + B); 

g * G/(R + G + B); and 

b - B/(R + G + B). 
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The saturation S can be represented by the following 
formula (2-3):- 

S = 1 - 3 rain fr, g, b) (2-3) 
where min (r, g, b) represents the minimum value of r, g and b. 

Thus, when the hue and saturation are made numerical 
values for respective pixels of the original picture, then just as 
in the case of noting the grey level, by a spatial filtering or the 
like, the hue and saturation variations can be extracted. Just as 
in the case of noting a grey level, by carrying out the second to 
fifth steps, the fold can be extracted by noting the colour. 

In a case where the original picture is given by an NTSC 
signal, the hue can be obtained from the phase of a chrominance 
signal and the saturation can be obtained from the amplitude of the 
chrominance signal. 

Also, the value of a specific colour component may be 

noted . 

In Figures 32 to 34, the second embodiment of the present 
invention is shown. 

In this embodiment, as compared with the first 
embodiment, the fourth step, that is the step of connecting contour 
segments is different. 

In this embodiment, a plurality of contour segments are 
simultaneously connected by using an overlapped pyramid. 

The overlapped pyramid structure used in this embodiment 
is shown in Figure 32. In this overlapped pyramid, respective 
nodes have 16 children nodes and 4 parent nodes. In Figure 32, 
4x4 (=16) nodes enclosed with fine solid state lines belong to 
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one parent node. That is to say, children nodes corresponding to a 
plurality of parent nodes are overlapped. Thus, one child node has 
4 parent nodes. Such an overlapped pyramid structure is made by 
considering a small region consisting of 8 x 8 pixels as a leaf 
node. 

Then, by using this overlapped pyramid structure, a 
method of connecting contour segments will be explained. 

In Figure 33, one parent node and 16 children nodes 
corresponding to the parent node in Figure 32 are parted and shown. 

First of all, in case there are a plurality of line 
segments in a region consisting of 2 x 2 {=4} nodes enclosed with 
thick solid lines among IS children nodes in Figures 32 and 33, it 
will be examined whether these segments can be connected. Here, 
the directions and edge points of two contour segments (which are 
hereinafter called connecting information) are considered to 
determine whether the two contour segments are connected. For 
example, if the difference of the direction between two contour 
segments is 45° or less, and also the distance between the edge 
points is a predetermined distance threshold Dmax or less, two 
contour segments are considered to be connected. 

In Figure 33, two sets of contour segments connected in 
accordance with the above mentioned rule are shown. In the example 
shown in this Figure, there are contour segments (a), (b), (c) and 
(d) within the respective 2x2 nodes. Since the difference in 
direction &i between contour segments {a} and (b) is 450 or 
less and also the distance Di between edge points is a distance 
threshold Dmax or less, they are considered to be connected as 
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the same group. Similarly, since the difference in direction 
©2 between the two contour segments (c) and (d) is 45° or 
less, and also the distance 02 between edge points is a distance 
threshold Dmax or less, they are considered to be connected as 
the same group. 

The coordinates (Y3, ¥4, Y5 and Y6 in Figure 33) 
of the edge points at which the two sets of lines connected in this 
manner meet the perimeter enclosing 4 children nodes (shown by 
thick solid lines), and the angles C © 3* ^ At ©5 and 
®6 in Figure 33) at which the two sets of lines intersect the 
perimeter, are raised to a parent node as connecting information of 
the parent node. 

Then, a plane to which the above mentioned parent node 
belongs is considered. That is, 4 parent nodes as shown in 
Figure 32 have the connecting information raised from their 
children nodes, respectively. In the regions consisting of these 4 
parent nodes, contour segments are connected in accordance with the 
above mentioned rule. 

The same is repeated up to the root level and the whole 
connection of contour segments is completed. 

The above mentioned operation will be explained by using 
a flow chart in Figure 34. 

First of all, a small region consisting of 8 x 8 pixels 
is made a leaf node in step S61. 

Then, in step S62, the coordinates of the points at which 
contour segments in respective nodes meet the perimeters of the 
nodes, 
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and the angles at which the contour segments and the perimeters 
intersect, are made connecting information. 

Next in step 363, when there are a plurality of contour 
segments in a region of 2 x 2 nodes and when the connecting 
information satisfies predetermined conditions, that is to say, 
when the difference in direction between two contour segments is 
450 or less, and also the distance between edge points is a 
predetermined distance threshold Dmax or less, contour segments 
are connected. 

Next, in step 564, a plane consisting of parent nodes 
having 4x4 overlapped children nodes is formed, and the 
coordinates of the points at which contour segments connected 
between 2x2 children nodes of the central part among 4x4 
children nodes meet perimeters enclosing 2x2 children nodes 
(corresponding to a 

perimeter of a parent node), and the angles at which the contour 
segments and the perimeters intersect, are made the connecting 
information in the parent node. 

Then in step S65, it is judged whether the number of 
nodes is one or not. If not, the process goes back to the above 
mentioned step S63 and steps S64 and 565 are carried out In the 
level of parent node. If the number of nodes is one, the process 
is completed. 

Thus, the formation of the overlapped pyramid and the 
connection of contour segments are carried out in parallel. The 
formation starts from a leaf level and when the root level is 
reached, the whole contour segments are connected. 
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The fourth step of the fixst embodiment is a method 
including the following steps. A fold is extracted by connecting 
contour segments one by one and the remaining contour segments are 
connected one by one in order to extract the next fold. However , 
in the fourth step of this embodiment, a plurality of folds are 
extracted In parallel so that processing time can be shortened. 

Also, according to the method of this embodiment, since a 
pyramid is overlapped and formed, the contour segments on a 
boundary of a region are not overlooked. 

Other information, operation and effects are the same as 
the first embodiment. 

The endoscope operator may insert the endoscope by the 
curving operation in the endoscope insertion direction detected by 
the method of the present Invention, or the endoscope may be 
inserted by automatically directing the tip part by using the 
apparatus in the manner described. 

As explained above, according to the present invention, 
by extracting the form of a fold and using a Judgement on the basis 
of this form of the fold, the endoscope Insertion direction can be 
simply detected. 

Also, by dividing an endoscope picture image into small 
regions In extracting contour segments and by utilising a pyramid 
quad tree structure in connecting the contour segments, there Is an 
effect of reducing the processing time. 
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In the case of extracting discontinuous points in an 
endoscope picture image, by setting the reference value to be 
rather low, extracting contour segment candidates and then 
extracting the optimum contour segment by a perceptual grouping, 
there are the effects that the influence of noise or the like can 
be reduced, the form of the fold can be extracted more accurately 
and the insertion direction can be detected. 

It is apparent that, in this invention, implementation 
methods differing over a vide range can be formed without departing 
from the spirit and scope of the invention. This Invention is not 
restricted by its specific working modes except being limited by 
the appended Claims. 
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PATENT CLAIMS: 

.1. A method of detecting a required endoscope insertion 

direction including the step of extracting the form of a fold 
existing on the inside wall of an observed part from an endoscope 
picture image so that the endoscope insertion direction may be 
judged on the basis of the form of the fold extracted by said step. 

2. A method according to Claim 1, wherein said fold form 
extracting step is to detect a part at which the brightness or 
colour of the picture image varies and to consider said part to be 
a fold. 

3. A method of detecting a required endoscope insertion 
direction comprising the respective steps of: 

extracting discontinuous points in an endoscope picture 

image; 

extracting contour segments based on said discontinuous 
points from respective divided picture Images obtained by dividing 
the picture image obtained by said discontinuous point extracting 
step into a plurality of picture images; 

connecting the contour segments obtained by extracting 
said contour segments; and 

considering the line connected by said connecting step to 
be the form of a fold existing on the inside wall of an observed 
part and judging the endoscope Insertion direction on the basis of 
said form of the fold. 
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4* A method according to Claim 3, wherein said discontinuous 

point extracting step includes extracting points at which the 
brightness of the picture Image varies. 

5. A method according to Claim 3, vherein said step of 
extracting discontinuous points includes extracting points at which 
the colour of the picture image varies. 

6. A method according to Claim 3, wherein said step of 
extracting discontinuous points includes extracting discontinuous 
points by applying a spatial filtering by using an overlapped 
matrix. 

7. A method according to Claim 3, vherein said step of 
extracting contour segments has a step of extracting contour 
segment candidates and a step of extracting the optimum contour 
segment from among the candidates extracted by said candidate 
extracting step. 

8. A method according to Claim 7, wherein said candidate 
extracting step includes using a modified Hough conversion in 
extracting the contour segments. 

9. A method according to Claim 7, vherein said modified 
Hough conversion includes defining the contour segments within the 
divided picture image by addresses attached to the periphery of the 
divided picture image. 
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10. A method according to Claim 7, wherein said optimum 
contour segment extracting step includes carrying out a perceptual 
grouping to extract the optimum contour segment. 

11. A method according to Claim 3, wherein said optimum 
contour segment extracting step utilises at least an edge 
orientation, grey level, continuity, edge magnitude or colour in 
carrying out said perceptual grouping. 

12. A method according to Claim 3, wherein said connecting 
step includes extracting a search starting contour segment by 
utilising a pyramid quad tree structure in connecting said contour 
segments . 

13. A method according to Claim 3, wherein said connecting 
step includes determining the next searched divided picture image 
in response to the arranging direction of the divided picture 
images having so far connected contour segments. 

14. A method according to Claim 3, wherein said connecting 
step includes limiting the angle formed by two connected contour 
segments to be withini45o in connecting said contour segments. 
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15. A method according to Claim 3, wherein, in a case where 

two carves having an intersection are obtained by said connecting 
step, said endoscope insertion direction judging step includes 
judging the direction of the contour segment at said intersection, 
thereby judging which curve is inner and then judging the centre of 
the inner curve to be the insertion direction* 

IS. A method according to Claim 3, wherein, in a case where a 

ring-like line is obtained by said connecting step, said endoscope 
insertion direction judging step includes judging the centre of 
gravity of said ring-like line to be the insertion direction. 

17. A method according to Claim 3, wherein said endoscope 
insertion direction judging step includes judging the part in which 
the intersections of radial lines erected respectively from a 
plurality of points on the curves obtained by said connecting step 
concentrate to be the required insertion direction. 

18. A method according to Claim 3, wherein, in a case where 
the curve obtained by said connecting step is like a partly Incised 
ring, said endoscope insertion direction judging step includes 
connecting the obtained curve at both ends and judging the centre 
of gravity of the thereby obtained ring to be the insertion 
direction. 
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19. A method according to Claim 3, wherein, in a case where 
the carve obtained by said connecting step is partly broken, said 
endoscope insertion direction judging step includes connecting the 
nearest curves and judging the centre of gravity of the obtained 
ring to be the insertion direction. 

20. A method according to Claim 1 or Claim 3, wherein said 
endoscope picture image is obtained by a television camera fitted 
to the endoscope eye-piece capable of naked eye observation. 

21. A method according to Claim 1 or Claim 3, wherein said 
endoscope picture image is obtained by an Imaging means provided in 
the endoscope. 

22. A method according to Claim 3, wherein said connecting 
step includes forming a pyramid of picture Images making picture 
linages consisting of an assemblage of said divided picture Images a 
bottom layer; and 

connecting the contour segments in order from the bottom 
layer of the pyramid of these Images to the root node. 

23. A method according to Claim 22, wherein a formation of 
said pyramid of picture Images and connection of contour segments 
are carried out in parallel in said connecting step. 

24. A method according to Claim 23, wherein said connecting 
step Includes connecting two contour segments for which the 
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coordinates of the point at which the contour segments In 
respective nodes and perimeters of the nodes meet, and the angles 
at which the contour segments intersect the perimeters, satisfy 
predetermined conditions among a plurality of children nodes 
belonging to a certain parent node in said pyramid of picture 
images. 

25. A method according to Claim 24, wherein said 

predetermined conditions are that the distance between said points 
is a predetermined distance or less and also the difference of said 
angles is a predetermined angle or less. 



26. A method according to Claim 24, wherein said pyramid of 
picture images is formed while the coordinates of the points at 
which the contour segments connected among a plurality of children 
nodes and lines corresponding to a perimeter of a parent node meet, 
and the angles at which the contour segments intersect perimeter 
lines, are made the connecting information of the parent node. 

27. A method according to Claim 22, wherein said pyramid of 
picture images has an overlapped pyramid structure in which 
respective children nodes corresponding to different parent nodes 
are overlapped. 



